{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12 Useful Pandas Techniques to add to your Arsenal!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "If you are reading this article, I'm sure you love statistical modeling (or will start loving it if you're a begineer)! Not sure about you, but during my initial days with machine learning, I was too anxious to make a model that I didn't even took the pain of even opening my data file and just ran my first model. As expected, this almost never worked out and my compiler started throwing errors. On backtracking, I found the most common reasons - missing data, incorrect coding of values, nominal variables being treated as numeric because the finite values are integers, etc.\n",
    "\n",
    "Data exploration and feature engineering are crucial for succssfully deriving actionable insights from data. In this article, I will share some tips and tricks using which this can be easily done using Pandas in Python. Python is gaining popularity and Pandas is one of the most popular tools for data handling in Python.\n",
    "\n",
    "Note: We'll be using the dataset of the \"Loan Prediction\" problem from the Analytics Vidhya datahacks. Click here to access the dataset and problem statement. The data can be loaded into Python using the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(614, 12)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Index([u'Gender', u'Married', u'Dependents', u'Education', u'Self_Employed',\n",
       "       u'ApplicantIncome', u'CoapplicantIncome', u'LoanAmount',\n",
       "       u'Loan_Amount_Term', u'Credit_History', u'Property_Area',\n",
       "       u'Loan_Status'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "data = pd.read_csv(\"train.csv\", index_col=\"Loan_ID\")\n",
    "# test = pd.read_csv(\"test.csv\", index_col=\"PassengerID\")\n",
    "print data.shape\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Boolean Indexing\n",
    "\n",
    "Useful for indexing a set of column using another set of columns. For instance, we can get a list of all women who are not graduate and got a loan using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>Education</th>\n",
       "      <th>Loan_Status</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Loan_ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LP001155</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001669</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001692</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001908</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002300</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002314</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002407</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002489</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002502</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002534</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002582</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002731</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002757</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002917</th>\n",
       "      <td>Female</td>\n",
       "      <td>Not Graduate</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Gender     Education Loan_Status\n",
       "Loan_ID                                   \n",
       "LP001155  Female  Not Graduate           Y\n",
       "LP001669  Female  Not Graduate           Y\n",
       "LP001692  Female  Not Graduate           Y\n",
       "LP001908  Female  Not Graduate           Y\n",
       "LP002300  Female  Not Graduate           Y\n",
       "LP002314  Female  Not Graduate           Y\n",
       "LP002407  Female  Not Graduate           Y\n",
       "LP002489  Female  Not Graduate           Y\n",
       "LP002502  Female  Not Graduate           Y\n",
       "LP002534  Female  Not Graduate           Y\n",
       "LP002582  Female  Not Graduate           Y\n",
       "LP002731  Female  Not Graduate           Y\n",
       "LP002757  Female  Not Graduate           Y\n",
       "LP002917  Female  Not Graduate           Y"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[(data[\"Gender\"]==\"Female\") & (data[\"Education\"]==\"Not Graduate\") & (data[\"Loan_Status\"]==\"Y\"), [\"Gender\",\"Education\",\"Loan_Status\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More: http://pandas.pydata.org/pandas-docs/stable/indexing.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Iterating over rows of a dataframe\n",
    "\n",
    "Although rarely used, but at times you may need to iterate through all rows. One common problem we face is nominal variables with numeric categories are treated as numerical by defauly by python. Also, there might be some numeric variable where some characters are entered in one of the rows and it'll be considered categorical by default. So it's generally a good idea to manually define the column types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                object\n",
       "Married               object\n",
       "Dependents            object\n",
       "Education             object\n",
       "Self_Employed         object\n",
       "ApplicantIncome        int64\n",
       "CoapplicantIncome    float64\n",
       "LoanAmount           float64\n",
       "Loan_Amount_Term     float64\n",
       "Credit_History       float64\n",
       "Property_Area         object\n",
       "Loan_Status           object\n",
       "dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Check current type:\n",
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that Credit_History is a nominal variable but appearing as float. A good way to tackle this issue is to create a csv file with column names and types. This way we can make a generic function to read the file and assign column data types. For instance, in this case I've defined a csv file datatypes.csv (download)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              feature         type\n",
      "0             Loan_ID  categorical\n",
      "1              Gender  categorical\n",
      "2             Married  categorical\n",
      "3          Dependents  categorical\n",
      "4           Education  categorical\n",
      "5       Self_Employed  categorical\n",
      "6     ApplicantIncome   continuous\n",
      "7   CoapplicantIncome   continuous\n",
      "8          LoanAmount   continuous\n",
      "9    Loan_Amount_Term   continuous\n",
      "10     Credit_History  categorical\n",
      "11      Property_Area  categorical\n",
      "12        Loan_Status  categorical\n"
     ]
    }
   ],
   "source": [
    "#Load the file:\n",
    "colTypes = pd.read_csv('datatypes.csv')\n",
    "print colTypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On loading this file, we can iterate through each row and assign the datatype from column 'type' to the variable name defined in 'feature' column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender                object\n",
      "Married               object\n",
      "Dependents            object\n",
      "Education             object\n",
      "Self_Employed         object\n",
      "ApplicantIncome        int64\n",
      "CoapplicantIncome    float64\n",
      "LoanAmount           float64\n",
      "Loan_Amount_Term     float64\n",
      "Credit_History       float64\n",
      "Property_Area         object\n",
      "Loan_Status           object\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "#Iterate through each row and assign variable type.\n",
    "# Note: astype is used to asign types\n",
    "\n",
    "for i, row in colTypes.iterrows():  #i: dataframe index; row: each row in series format\n",
    "    if row['feature']==\"categorical\":\n",
    "        data[row['feature']]=data[row['feature']].astype(np.object)\n",
    "    elif row['feature']==\"continuous\":\n",
    "        data[row['feature']]=data[row['feature']].astype(np.float)\n",
    "print data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the credit history column is modified to 'object' type which is used for representing nominal variables in Pandas.\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iterrows.html#pandas.DataFrame.iterrows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Apply Function\n",
    "\n",
    "It is one of the most handy functions when it comes to playing with the data, specially creating new varaibles. As the name suggests, apply returns some value after passing each row/column of a dataframe through some function. The function can be both default or user-defined.\n",
    "\n",
    "For instance, here it can be used to find the #missing values in each row and column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values per column:\n",
      "Gender               13\n",
      "Married               3\n",
      "Dependents           15\n",
      "Education             0\n",
      "Self_Employed        32\n",
      "ApplicantIncome       0\n",
      "CoapplicantIncome     0\n",
      "LoanAmount           22\n",
      "Loan_Amount_Term     14\n",
      "Credit_History       50\n",
      "Property_Area         0\n",
      "Loan_Status           0\n",
      "dtype: int64\n",
      "\n",
      "Missing values per row:\n",
      "Loan_ID\n",
      "LP001002    1\n",
      "LP001003    0\n",
      "LP001005    0\n",
      "LP001006    0\n",
      "LP001008    0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Create a new function:\n",
    "def num_missing(x):\n",
    "    return sum(x.isnull())\n",
    "\n",
    "#Applying per column:\n",
    "print \"Missing values per column:\"\n",
    "print data.apply(num_missing, axis=0)  #axis=0 defines that function is to be applied on each column\n",
    "\n",
    "#Applying per row:\n",
    "print \"\\nMissing values per row:\"\n",
    "print data.apply(num_missing, axis=1).head()  #axis=1 defines that function is to be applied on each column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus we get the desired result. Note: head() function is used in second output because it contains many rows. \n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.apply.html#pandas.DataFrame.apply"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Imputing missing files\n",
    "\n",
    "'fillna()' is a very useful function handy in implementing very basic forms of imputation like updating missing values with the overall mean/mode/median of the column. Let's impute the 'Gender', 'Married' and 'Self_Employed' columns with there respective modes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/aarshay/anaconda/lib/python2.7/site-packages/numpy/lib/arraysetops.py:200: FutureWarning: numpy not_equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (`is`)) and will change.\n",
      "  flag = np.concatenate(([True], aux[1:] != aux[:-1]))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "ModeResult(mode=array(['Male'], dtype=object), count=array([489]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#First we import a function to determine the mode\n",
    "from scipy.stats import mode\n",
    "mode(data['Gender'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns both mode and count. Remember that mode can be an array as there can be multiple values with high frequency. We will take the first one by default always using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Male'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mode(data['Gender']).mode[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can fill the missing values and check using technique #3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender                0\n",
      "Married               0\n",
      "Dependents           15\n",
      "Education             0\n",
      "Self_Employed         0\n",
      "ApplicantIncome       0\n",
      "CoapplicantIncome     0\n",
      "LoanAmount           22\n",
      "Loan_Amount_Term     14\n",
      "Credit_History       50\n",
      "Property_Area         0\n",
      "Loan_Status           0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Impute the values:\n",
    "data['Gender'].fillna(mode(data['Gender']).mode[0], inplace=True)\n",
    "data['Married'].fillna(mode(data['Married']).mode[0], inplace=True)\n",
    "data['Self_Employed'].fillna(mode(data['Self_Employed']).mode[0], inplace=True)\n",
    "\n",
    "#Now check the #missing values again to confirm:\n",
    "print data.apply(num_missing, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence confirmed the missing values are imputed. Note: This is the most primitive forms of imputation. Other sophisticated techniques include modeling the missing values using some variables or using grouped averages (mean/mode/median). We'll take up the later next.\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html#pandas.DataFrame.fillna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Pivot Table\n",
    "\n",
    "Pandas can be used to create MS Excel style pivot tables. For isntance, in this case a very key column is \"LoanAmount\" which has missing values. We can impute it using mean amount of each 'Gender', 'Married' and 'Self_Employed' group. The mean 'LoanAmount' of each group can be determined as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                              LoanAmount\n",
      "Gender Married Self_Employed            \n",
      "Female No      No             114.691176\n",
      "               Yes            125.800000\n",
      "       Yes     No             134.222222\n",
      "               Yes            282.250000\n",
      "Male   No      No             129.936937\n",
      "               Yes            180.588235\n",
      "       Yes     No             153.882736\n",
      "               Yes            169.395833\n"
     ]
    }
   ],
   "source": [
    "#Determine pivot table\n",
    "impute_grps = data.pivot_table(values=[\"LoanAmount\"], index=[\"Gender\",\"Married\",\"Self_Employed\"], aggfunc=np.mean)\n",
    "print impute_grps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pivot_table.html#pandas.DataFrame.pivot_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. MultiIndexing\n",
    "\n",
    "If you notice the output of step #4, it has a peculiar property. Each index is made of a combination of 3 values! This is called MultuIndexing and it can be used to perform some operations really fast if used wisely. Continuing the example from #4, we have the values for each group but not they have to be imputed.\n",
    "\n",
    "This can be done using the various techniques learned till now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender                0\n",
      "Married               0\n",
      "Dependents           15\n",
      "Education             0\n",
      "Self_Employed         0\n",
      "ApplicantIncome       0\n",
      "CoapplicantIncome     0\n",
      "LoanAmount            0\n",
      "Loan_Amount_Term     14\n",
      "Credit_History       50\n",
      "Property_Area         0\n",
      "Loan_Status           0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#iterate only through rows with missing LoanAmount\n",
    "for i,row in data.loc[data['LoanAmount'].isnull(),:].iterrows():\n",
    "    ind = tuple([row['Gender'],row['Married'],row['Self_Employed']])\n",
    "    data.loc[i,'LoanAmount'] = impute_grps.loc[ind].values[0]\n",
    "\n",
    "#Now check the #missing values again to confirm:\n",
    "print data.apply(num_missing, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "    1. Multi-index require tuple for defining groups of indices in loc statement. This a tuple used in function.\n",
    "    2. The .values[0] suffix is required because by defualt a series element is returned which has an index not matching with that of the dataframe. So direct assignment gives an error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Crosstab\n",
    "\n",
    "This function can be used to get an initial \"feel\" of the data. We can validate some basic hypothesis. For instance, in this case \"Credit_Histpry\" is expected to affect the loan status significantly. This can be tested using a cross-tabulation as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Loan_Status</th>\n",
       "      <th>N</th>\n",
       "      <th>Y</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Credit_History</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>82</td>\n",
       "      <td>7</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>97</td>\n",
       "      <td>378</td>\n",
       "      <td>475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>192</td>\n",
       "      <td>422</td>\n",
       "      <td>614</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Loan_Status       N    Y  All\n",
       "Credit_History               \n",
       "0.0              82    7   89\n",
       "1.0              97  378  475\n",
       "All             192  422  614"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(data[\"Credit_History\"],data[\"Loan_Status\"],margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are absolute numbers but percentages can be more intuitive in making some quick insights. We can do this using the apply function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Loan_Status</th>\n",
       "      <th>N</th>\n",
       "      <th>Y</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Credit_History</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>0.921348</td>\n",
       "      <td>0.078652</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.204211</td>\n",
       "      <td>0.795789</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>0.312704</td>\n",
       "      <td>0.687296</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Loan_Status            N         Y  All\n",
       "Credit_History                         \n",
       "0.0             0.921348  0.078652    1\n",
       "1.0             0.204211  0.795789    1\n",
       "All             0.312704  0.687296    1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def percConvert(ser):\n",
    "    return ser/float(ser[-1])\n",
    "pd.crosstab(data[\"Credit_History\"],data[\"Loan_Status\"],margins=True).apply(percConvert, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is clearly evident that people with a credit histpry have much higher chances of getting a loan as 80% people with credit history got a loan as compared to only 9% without credit histoty. \n",
    "\n",
    "But that's not it. It tells an iteresting story. Since I know that having a credit history is super important, what if I predict loan status to be Y for ones with credit history and N otherwise. Surprisingly, we'll be right 82+378=460 times out of 614 which is a whopping 75%! I won't blame you if you're wondering why the hell do we need statistical models. But trust me, increasing the accuracy by even 0.001% beyond this is a challenging ask.\n",
    "\n",
    "Note: 75% is on train set. The test set will be slightly different but close.\n",
    "Also, I hope this gives some intuition into why even a 0.05% increase in accuracy can result in jump of 500 ranks on a Kaggle leaderboard.\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.crosstab.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Merge DataFrames\n",
    "\n",
    "Merging dataframes becomes essential when we have information coming from different sources which is to be collated. Consider a hypothetical case where the average property rates (INR per sq meters) is available for different property types. Let's define a dataframe as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Rural</th>\n",
       "      <td>1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Semiurban</th>\n",
       "      <td>5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Urban</th>\n",
       "      <td>12000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           rates\n",
       "Rural       1000\n",
       "Semiurban   5000\n",
       "Urban      12000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prop_rates = pd.DataFrame([1000, 5000, 12000], index=['Rural','Semiurban','Urban'],columns=['rates'])\n",
    "prop_rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can merge this information with the original dataframe as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Property_Area  rates\n",
       "Rural          1000     179\n",
       "Semiurban      5000     233\n",
       "Urban          12000    202\n",
       "Name: Credit_History, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_merged = data.merge(right=prop_rates, how='inner',left_on='Property_Area',right_index=True, sort=False)\n",
    "data_merged.pivot_table(values='Credit_History',index=['Property_Area','rates'], aggfunc=len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pivot table validates sucessful merge operation. Note that the 'values' argument is irrelevant here because we are simply counting the values.\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html#pandas.DataFrame.merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Sorting DataFrames\n",
    "\n",
    "Pandas allows easy sorting based on multiple columns. This can be done as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ApplicantIncome</th>\n",
       "      <th>CoapplicantIncome</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Loan_ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LP002317</th>\n",
       "      <td>81000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002101</th>\n",
       "      <td>63337</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001585</th>\n",
       "      <td>51763</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001536</th>\n",
       "      <td>39999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001640</th>\n",
       "      <td>39147</td>\n",
       "      <td>4750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002422</th>\n",
       "      <td>37719</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001637</th>\n",
       "      <td>33846</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001448</th>\n",
       "      <td>23803</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP002624</th>\n",
       "      <td>20833</td>\n",
       "      <td>6667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LP001922</th>\n",
       "      <td>20667</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          ApplicantIncome  CoapplicantIncome\n",
       "Loan_ID                                     \n",
       "LP002317            81000                  0\n",
       "LP002101            63337                  0\n",
       "LP001585            51763                  0\n",
       "LP001536            39999                  0\n",
       "LP001640            39147               4750\n",
       "LP002422            37719                  0\n",
       "LP001637            33846                  0\n",
       "LP001448            23803                  0\n",
       "LP002624            20833               6667\n",
       "LP001922            20667                  0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_sorted = data.sort_values(['ApplicantIncome','CoapplicantIncome'], ascending=False)\n",
    "data_sorted[['ApplicantIncome','CoapplicantIncome']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The Pandas \"sort\" function is now depricated and we should use \"sort_values\".\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_values.html#pandas.DataFrame.sort_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10. Plotting (Boxplot & Histogram)\n",
    "\n",
    "Many of you might be unaware that boxplots and histograms can be directly plotted in Pandas and calling matplotlob separately is not necessary. It's just a 1-line command. For instance, if we want to compare the distribution of ApplicantIncome by Loan_Status:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10a4d1bd0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEbCAYAAAArhqjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2cXVV97/HPl2AAgWQSlURDIGoSQBGG0EKttImlBbEW\ngg8xaAuR2L5qUEJ9uCZ6bYZWXxp6tZFS8GopGSIYI16pFUwCwkCtBCIQgoIQihPyYAYhBHzEJPzu\nH2udw85kzpwzk5lz5mS+79frvNh7nbX3WWfYOb+9nvZSRGBmZtabAxpdADMzG/ocLMzMrCoHCzMz\nq8rBwszMqnKwMDOzqhwszMysKgcLMzOrysHCeiVpt6T7JK2T9ENJfzAIn/GLKu8fLem8gf7cwSbp\nAkn/0kP6Ikkf7uc5j5b04L6Xrk+fKUlflPSgpPWS7pZ0dH5vYY3nqCmfDV0OFlbNryJiWkS0Ap8A\nPjcIn1FtZuirgffsywdIatS1PhizXus9k/bdwCsj4g0RcQJwLrAjv/eJGs9Raz4bohwsrBoVtkcD\n28tvSP+U7zYfkDQrp82UdGvefqWkRyQdke+yb5R0e077+x4/bM9zvisnfxY4Lddw5nfLL0lXSnpI\n0ipJN0l6e37vp5I+J+mHwDslnSjprlxL+qak0Tnf7ZKm5e2XSfpp3q5YZknvzXfY90m6SpJy+vty\n3jXAm3r5u7ZK+kHOOzcf2y7p7MJnfFXSX/RyjuLfodJ3e7+keyTdL+kbkg7O6dfk2sJ/S3qs9Der\n4JXAz0o7EbE1Ip6V9FngkPw3WJbP+y1Ja/P/w/fntD3yda8dSfpI6W8r6WJJP87f4/pavrvVSUT4\n5VfFF7ALuA94GHgGOCmnvx1YlbePADYC4/L+tcBFwH8Cs3LaBcAWoAU4GHgQmJbfey7/9x09nROY\nDny7QvneAXwnb48jBbO35/2fAh8t5H0AOC1vXwp8IW/fXijLy4DHeyszcCzwbWBEzvevwF8C43OZ\nxwIHAt8HLu+hzIuA+4GR+fOeyMf+MfCtnGcU8D/AAd2OPRpY38M5u3+3f87bYwp5/hG4KG9fA3w9\nbx8HbOjlGpiQ/5b3Af8HaC2891y3vC35v6W/15ju+bp/B+AjwN/n7S3AS0p/g0Zf/369+HLNwqr5\ndaRmqOOAs4BlOf004GsAEfEk0AH8fn7vYmAh8NuIWFE41y0RsSMifgv8v3yOojf1cs5KTgO+kY/p\nIv3wF30dQNIoYHREfD+nt5N+nKsplvmb+fNOB04G1kq6H/gT4DXAqcDtEbE9InaVPruC/4iI30XE\n08BtwCkRcScwWdLLgPOAb0bEC9UKWOG7/VHePkHSnZLWk5ryXl849EaAiHiYFJx7FBFbgKmk/6cv\nALdKenOF7JdIWgesAY4EplQrfzcPANdLei+wu4/H2iA6sNEFsOYREWskvVzSy3t4u9hcNZH0ozKu\n+ymq7Pd2zv76VQ15dvFik+zB3d4rllGF/aUR8cliRknnUHuZK533WuCvgNnAnBrP1ZtrgLMj4keS\nLiDV0kqe71aGiiJiJ7AKWCWpC5hJCszl4yRNJwXOUyPieUm38+Lfs3j+XcCIwn7xb/7npCB+NvBJ\nScfXEjBt8LlmYdUUfwyOJV0zTwP/Bbxb0gGSXkG6k71H0oHA1aQfu4clfaRwrj+T1CLpENKPzfe7\nfUaP5wR+ARxeoXz/Dbwj912MA2b0lCkingOekVTqR/gr4I683Qn8Xt5+V7dDu5f5v0k1gXfmMiJp\njKSjgLuBP877L+nhXEXnSBqZaxHTgbU5vR24JBU5flLh2D1+2Ct8t468fRiwLZfnvb2Up2KwkHSS\npFfm7QOAE0h/M4DfSSr98I8GnsmB4ligOHLud/naAOgCXpH/TgcBbyvkOyoi7gAWkJriDuulzFZH\nrllYNQdLuo8Xf0zOj4gAvqU0jPYBUi3iYxHxpKRPAXdGxA9y08c9kr6Tj72H1Pw0AVgWEffn9ACI\niErn3A68kJt8lkbEFwvl+ybpbvbHwCbgXuDZ4nkLLgD+r6RXAYcCr83p5wFvkvTXwE3djule5vsA\nJP1vYHX+8fwdqS/gHkltpCaYZ4B1vfxd15N+0F8G/ENEbMt/gyclPQx8q5djp0p6ghdrJH9X+G6H\nAI8D78t5P5W/w5OkYFYKun2p5R0BfEXSyLx/D6mfBuDLwIOS7gUuBP5W0o+BR4C7Cuf4MrBe0r0R\n8VeS/pEUIDeT+sPIweSruVlNwBdzILQhQOnfvdngyk0gJ0fExf04toN0NzsuN4d0f//QiPiVpLGk\nH8Q35T6P3soyNyJ67bPYlzL3cK4XgMkR8Xjenw58NSImdsv3UlKwnBYRvc4/MasnN0PZkKY0+es0\nUk3j7ArZvpNrHXeS7tIrBooG6umubI80SacDD5FGUDlQ2JDiYGF1ERHt/bxDP5/UnLGUQodvnidw\nlaTVpJFJO4C3RkRpvP8Lkj4k6X8kPSnpskofkPO+Jm8fLOnzkjqBJaT5EAfl91ZI+pmkZyR1SHpd\nt/JcIek7kp5TmvPw6vzeHaRmlfX5vb36MnJn8HRSs8xnJK3MNaXS+6cpzYl4RtJGSefn9FGSrs3f\n8aeSPlk45gJJ35f0hXzcY5LemNOfkLStdJ6cv1VSl6TnJe2U9HOl+SJmDhY25J0PfBW4Hjiz1Kmc\nvYc0p+BlpKab67odO5M0L2IaqUP5wgqfUbzD/zxwEqlzdizwv0i1GoCbSf0cR5DmHHT/vHeT5lC0\nkOZIfAYgIkojkN4QEaMi4hsVynEeqe/hFcBBwEehXLu6Gfgi8HKglRf7Q64g9UNMInXuny/pfYVz\nnpLzjiUNS15O6sx/Lakj/Irc9EX+7Lvy92sBfgB8r0JZbbhp9EQPv/yq9CI1Pz3PixO7HgLm5+1r\ngOsLeQ8lDcmckPdfAP6s8P4HSHMmIP0o3ll47wXSPAkBvwaOr6FsLfm4wwvl+XLh/bOAh7p/RmF/\nOvBEYf924BPdyntz3l5AmnPRvQwH5L/PMYW0vwFuK3zPRwrvHU+au/DyQtpTwAl5+5fAqwvvvZE8\nQdEvv1yzsKHsfGB1RDyT979G+gEs2VTaiIhfkWZvv6rw/ubC9sZu7/Xk5aQ7+se7v5GH834uN+Xs\nIM1ojnxMybbC9q/p+7DPSsdPJNVUeirvgaQZ4CUbSSO3SroK278BiIinuqUdlmtsLwXulbQ9j0D7\nLqnWZuahszY0KT3DaBZwgKTSc4kOAkZLOiHvTyzkP4zU1LKlcJqJ5GGZwFHA1iof+xTwW1ITTfcn\nu74H+AvgTyLiCaVnLz3DwEwcrGYTqTmpu6eAnaTHZ5TmZBzNnn+DWj1FClCvj4ifVctsw49rFjZU\nnUtqVjoOODG/jiVN3Ct1yr5V0h/m8f//CNwVEcWA8DGlCXUTgfmk9vqKIiJIzUlfUHoI4gGS/iCf\n/3BSk88zkg4lPdywL+POt5GauvrjOuB0Se+UNELSWEknRprZvILUIX5Y7tv4O158JEtPegxu+bt/\nBViiFycbTpB0Rj/LbPsZBwsbqs4H/j0itkTEk6UXaTLYe0iPi7geaCPNKD+J9DC/ov8gTdK7j/RQ\nw3+v8FnFH/2PkmoVa/N5P0f6gb2W1NyzBfgRqfO3L9qAa3MTzzurlGHPNyI2AW/NZdtOeghhqXZ1\nMalG8Dhp6PBXI+KaXsrR22S8BcBjwJrc1Laa9Ewos9om5Sk9Fvr9efcrEXG5pDGkB6UdTZr6Pysi\nns35F5Jmc+4idUiuzunTSEMgDyZ13l2S00eS/jGeTKoOvzsiiu2wZnuQdA2wKSIqPep8j0lwZrZv\nqtYsJL0emEsabtcKvE3Sa0l3IbdGxDGkZ+UszPlfR2prLj2l9EpJparvVaSZs1NJjyw4M6fPBbZH\nxBTS2PaKY+LNzKz+ammGOg64OyKej4jdpKru20mzadtznnbSmHZy+vKI2BURncAG4BRJ40nDDEsP\nTLu2cMw5hXPdQHoEtFlvqlWJ/RwbswFUy2ioHwGfzs1Oz5PaTn9Iek5PF0BEbJNUeh7+BPZ8gNiW\nnLaLPYcybubFIX4TyMMgI2K3pB2SxkbEdsx6EBGVJtiV3h/R2/tm1jdVg0VE/ETSYuAW0qSd++l5\nUZKBvJOrx3BEMzOrUU3zLPLoimsAJH2GVAvokjQuIrpyE1Pp4W1bKIx/J62WtaWX9OIxW5WejT+q\np1qFJDctmJkNsojY64a9pqGzhXHXR5HGv19PWoN4Ts5yAWmYIjl9ttLCLq8GJgP3RHpe/7OSTskd\n3ud3O6Y0M/ddpA7zSl/CrwF+LVq0qOFl8Muvvrx8zQ7eq5JaZ3B/Mz8BcycwLyKey01TK/LD2TaS\nRkAREQ9JWkF6jk8pf6kEF7Hn0NmVOf1qYJmkDaSx7bNrLJeZmdVBrc1Qey0SE6mZ6E8r5P8saYZr\n9/R7gTf0kP48OdhY/XV2dja6CGZ94mu2/jyD22htbW10Ecz6xNds/TXVsqqSopnKa2bWbCQR/e3g\nNjOz4c3Bwujo6Gh0Ecz6xNds/TlYmJlZVe6zMDOzMvdZmJlZvzlYmNt/ren4mq0/BwszM6vKfRZm\nZlbmPgszM+s3Bwtz+681HV+z9edgYWZmVbnPwszMytxnYRW5Sm9m1dS6Ut7fSfqRpPWSrsur4I2R\ntFrSI5JWSRpdyL9Q0gZJD0s6o5A+LZ/jUUlLCukjJS3Px9yVV+SzOlm6dGmji2DWJ77Bqb+qwULS\nq4APAdMi4gTSgknnAQuAWyPiGNIyqAtz/teRFjI6DjgLuDIvowpwFTA3IqYCUyWdmdPnAtsjYgqw\nBLhsgL6fmZkNgFqXVR0BHCrpBeAQYAspOEzP77cDHaQAcjawPCJ2AZ15qdRTJG0EDo+ItfmYa4GZ\nwCrgHGBRTr8BuGJfvpRV19HRUb47a29vZ9KkSQDMmDGDGTNmNKxcZrXwNVp/VYNFRGyV9HngCeDX\nwOqIuFXSuIjoynm2SToiHzIBuKtwii05bRewuZC+OaeXjtmUz7Vb0g5JY/PSrTYIugeFtra2hpXF\nzIa+WpqhWkh3/kcDryLVMN4LdB+WNJDDlPbqibfB4/WMrdm4z6L+ammG+lPg8dJdvqRvAX8IdJVq\nF5LGA0/m/FuAiYXjj8xpldKLx2yVNAIYValWMWfOnHKTSUtLC62treU75NIF5P2+7ZfWMx4q5fG+\n971fv/3SdrWbxqrzLCSdAlwN/D7wPHANsBY4itQpvVjSx4ExEbEgd3BfB5xKal66BZgSESFpDXBx\nPv4m4PKIWClpHnB8RMyTNBuYGRGzeyiL51mYmQ2iSvMsaumzuEfSDcD9wM783y8DhwMrJF0IbCSN\ngCIiHpK0Ango559X+IW/CFgKHAzcHBErc/rVwLLcGf40sFegMDOzxvEMbqOjo6NcNTVrBr5mB49n\ncJuZWb+5ZmFmZmWuWZiZWb85WNgeQ+jMmoGv2fpzsDAzs6rcZ2FmZmXuszAzs35zsDC3/1rT8TVb\nfw4WZmZWlfsszMyszH0WZmbWbw4W5vZfazq+ZuvPwcLMzKpyn4WZmZW5z8LMzPqtljW4p0q6X9J9\n+b/PSrpY0hhJqyU9ImmVpNGFYxZK2iDpYUlnFNKnSVov6VFJSwrpIyUtz8fcJemogf+qVonbf63Z\n+Jqtv6rBIiIejYiTImIacDLwK+BbwALg1og4BrgNWAiQl1WdBRwHnAVcKalUpbkKmBsRU4Gpks7M\n6XNJS7ROAZYAlw3UFzQzs33Xpz6LXEv4VET8kaSfANMjokvSeKAjIo6VtACIiFicj/ku0EZaevW2\niHhdTp+dj/+ApJXAooi4W9IIYFtEvKKHz3efhZnZIBqoPot3A9fn7XER0QUQEduAI3L6BGBT4Zgt\nOW0CsLmQvjmn7XFMROwGdkga28eymZnZIKk5WEh6CXA28I2c1P0WfyBv+feKajZ43P5rzcbXbP0d\n2Ie8ZwH3RsRTeb9L0rhCM9STOX0LMLFw3JE5rVJ68ZituRlqVERs76kQc+bMYdKkSQC0tLTQ2tpa\nXri9dAF5v2/7JUOlPN73vvfr+++/o6ODzs5OelNzn4WkrwErI6I97y8mdUovlvRxYExELMgd3NcB\np5Kal24BpkRESFoDXAysBW4CLo+IlZLmAcdHxLzclzEzImb3UAb3WZiZDaJKfRY1BQtJLyV1UL8m\nIn6R08YCK0g1go3ArIjYkd9bSBrhtBOYHxGrc/rJwFLgYODmiJif0w8ClgEnAU8DsyOis4dyOFiY\nmQ2ifQoWQ4WDxeDo6OgoV03NmoGv2cHjGdxmZtZvrlmYmVmZaxZmZtZvDha2xxA6s2bga7b+HCzM\nzKwq91mYmVmZ+yzMzKzfHCzM7b/WdHzN1p+DhZmZVeVgYZ4Ja2ZVOViYWdNxM1T9OViY/+FZ06n2\nOG0beH1Zz8LMrGE6OjrKNzbt7e3ldW1mzJjhptQ68DwLM2s6bW1ttLW1NboY+yXPszAzs36rKVhI\nGi3pG5IelvRjSadKGiNptaRHJK2SNLqQf6GkDTn/GYX0aZLWS3pU0pJC+khJy/Mxd0k6amC/pvXG\nfRbWbFpaWhpdhGGn1prFF0kr2x0HnAj8BFgA3BoRxwC3AQsB8rKqs4DjSOt2XympVKW5CpgbEVOB\nqZLOzOlzSUu0TgGWAJft8zczs/1Wa2tro4sw7FTts5A0Crg/Il7bLf0nwPSI6JI0HuiIiGMlLQAi\nIhbnfN8F2khLr94WEa/L6bPz8R+QtBJYFBF3SxoBbIuIV/RQFvdZmJkNon3ps3g18JSkayTdJ+nL\neU3ucRHRBRAR24Ajcv4JwKbC8Vty2gRgcyF9c07b45iI2A3syGt8m5nZEFBLsDgQmAb8a0RMA35F\naoLqfos/kLf8e0U1Gzzus7Bm42u2/mqZZ7EZ2BQRP8z73yQFiy5J4wrNUE/m97cAEwvHH5nTKqUX\nj9mam6FGRcT2ngozZ86c8vjqlpYWWltby2OsSxeQ9/u2XzJUyuN973u/vv/+Ozo6qk50rGmehaQ7\ngL+OiEclLQJemt/aHhGLJX0cGBMRC3IH93XAqaTmpVuAKRERktYAFwNrgZuAyyNipaR5wPERMS/3\nZcyMiNk9lMN9FmZmg6hSn0WtweJE4N+AlwCPA+8DRgArSDWCjcCsiNiR8y8kjXDaCcyPiNU5/WRg\nKXAwaXTV/Jx+ELAMOAl4GpgdEZ09lMPBwsxsEO1TsBgqHCwGR0dHR7lqatYMfM0OHs/gNjOzfnPN\nwszMylyzMDOzfnOwsD2G0Jk1A1+z9edgYWZmVbnPwszMytxnYWZm/eZgYW7/tabja7b+HCzMzKwq\n91mYmVmZ+yzMzKzfHCzM7b/WdHzN1p+DhZmZVeVgYX56p5lV5WBhZk3HzVD1V1OwkNQp6QFJ90u6\nJ6eNkbRa0iOSVkkaXci/UNIGSQ9LOqOQPk3SekmPSlpSSB8paXk+5i5JRw3kl7Te+R+eNZtqS4Da\nwKtlDW6AF4AZEfFMIW0BcGtEXJaXVV0IlJZVnQUcR1pn+1ZJU/KY16uAuRGxVtLNks6MiFWkVfW2\nR8QUSe8GLgP2WlbVzIavjo6O8o1Ne3s7kyZNAlIzqptSB1+ty6r+FPi9iHi6kPYTYHpEdEkaD3RE\nxLGSFgAREYtzvu8CbaSlV2+LiNfl9Nn5+A9IWgksioi7JY0AtkXEK3ooh+dZmBltbW20tbU1uhj7\npX2dZxHALZLWSnp/ThsXEV0AEbENOCKnTwA2FY7dktMmAJsL6Ztz2h7HRMRuYIeksTWWzczMBlmt\nweJNETENeCtwkaQ/IgWQooG85d8rqtngcZ+FNZuWlpZGF2HYqanPIiJ+lv/7c0k3AqcAXZLGFZqh\nnszZtwATC4cfmdMqpReP2ZqboUZFxPaeyjJnzpxyW2VLSwutra3l9srSj573+7ZfMlTK433vV9tv\nbW0dUuVp5v3SdrVBA1X7LCS9FDggIn4p6VBgNXApcDqpU3px7uAeExGlDu7rgFNJzUu3AFMiIiSt\nAS4G1gI3AZdHxEpJ84DjI2Je7suYGRF7dXC7z8LMbHBV6rOopWYxDviWpMj5r4uI1ZJ+CKyQdCGp\n83oWQEQ8JGkF8BCwE5hX+IW/CFgKHAzcHBErc/rVwDJJG4Cn8UgoM7MhxU+dNTo6OspVU7Nm4Gt2\n8Pips2Zm1m+uWZiZWZlrFmZm1m8OFrbHEDqzZuBrtv4cLMzMrCr3WZiZWZn7LMzMrN8cLMztv9Z0\nfM3Wn4OFmZlV5T4LMzMrc5+FmZn1m4OFuf3Xmo6v2fpzsDAzs6rcZ2FmZmXuszAzs36rOVhIOkDS\nfZK+nffHSFot6RFJqySNLuRdKGmDpIclnVFInyZpvaRHJS0ppI+UtDwfc5ekowbqC1p1bv+1ZuNr\ntv76UrOYT1r9rmQBcGtEHAPcBiwEyMuqzgKOA84CrpRUqtJcBcyNiKnAVEln5vS5pCVapwBLgMv6\n+X3MzGwQ1BQsJB0JvBX4t0LyOUB73m4HZubts4HlEbErIjqBDcApksYDh0fE2pzv2sIxxXPdQFrf\n2+rEK46ZWTW11iz+GfgYUOxdHhcRXQARsQ04IqdPADYV8m3JaROAzYX0zTltj2MiYjewQ9LY2r+G\nmQ0nboaqv6rBQtKfA10RsQ7Yq4e8YCCHKfX2OTbA/A/Pms2aNWsaXYRh58Aa8rwJOFvSW4FDgMMl\nLQO2SRoXEV25ienJnH8LMLFw/JE5rVJ68ZitkkYAoyJie0+FmTNnDpMmTQKgpaWF1tbWcjNK6UfP\n+33bLxkq5fG+93vaX7JkCevWrWPSpEmsWrWKOXPmAOk3YcaMGQ0vX7Pul7Y7OzvpTZ/mWUiaDnwk\nIs6WdBnwdEQslvRxYExELMgd3NcBp5Kal24BpkRESFoDXAysBW4CLo+IlZLmAcdHxDxJs4GZETG7\nh8/3PAsz2yM42MCqNM+ilppFJZ8DVki6ENhIGgFFRDwkaQVp5NROYF7hF/4iYClwMHBzRKzM6VcD\nyyRtAJ4G9goUZja8dXR0lAPEHXfcQVtbG5ACR+lu2QaPZ3AbHR0d/sdmTWXy5Mk89thjjS7Gfskz\nuM1sv3HYYYc1ugjDzr40Q9l+wrUKawbFZqgHHnjAzVB15mBhZk2he1AoBQurDzdDmUeVWNOpNszT\nBp6DhZk1ndbW1kYXYdjxaCgzMyvzaCgzM+s3Bwtzn4U1HV+z9edgYaxbt67RRTCzIc7BwtixY0ej\ni2DWJ55XUX8OFmZmVpUn5Q1Txdmwl156aTnds2GtGfh5ZvXnYDFMFYNCZ2enZ8OaWa/cDGXlxaTM\nmoVrFfXnYGH+h2dmVdWyBvdBku6WdL+kByUtyuljJK2W9IikVZJGF45ZKGmDpIclnVFInyZpvaRH\nJS0ppI+UtDwfc5ekowb6i5rZ/sPzLOqvarCIiOeBN0fESUArcJakU4AFwK0RcQxwG7AQIC+rOgs4\nDjgLuFJSaer4VcDciJgKTJV0Zk6fC2yPiCnAEuCygfqCZma272pqhoqIX+fNg0id4gGcA7Tn9HZg\nZt4+G1geEbsiohPYAJwiaTxweESszfmuLRxTPNcNwOn9+jbWL26Gsmbja7b+agoWkg6QdD+wDbgl\n/+CPi4gugIjYBhyRs08ANhUO35LTJgCbC+mbc9oex0TEbmCHpLH9+kZmZjbgaq1ZvJCboY4k1RJe\nT6pd7JFtAMu11xMPbfC4/deaja/Z+uvTPIuIeE5SB/AWoEvSuIjoyk1MT+ZsW4CJhcOOzGmV0ovH\nbJU0AhgVEdt7KsOcOXPKQz1bWlpobW0tV0lLF5D3+7ZfMlTK433ve7++//47OjqqLihVdT0LSS8H\ndkbEs5IOAVYBnwOmkzqlF0v6ODAmIhbkDu7rgFNJzUu3AFMiIiStAS4G1gI3AZdHxEpJ84DjI2Ke\npNnAzIiY3UNZvJ6FmdkgqrSeRS01i1cC7ZIOIDVbfT0ibs4//CskXQhsJI2AIiIekrQCeAjYCcwr\n/MJfBCwFDgZujoiVOf1qYJmkDcDTwF6BwszMGscr5Rkdfs6ONRlfs4PHK+WZmVm/uWZhZmZlrlmY\nmVm/OVjYHkPozJqBr9n6c7AwM7Oq3GdhZmZl7rMwM7N+c7Awt/9a0/E1W38OFmbWdNatW9foIgw7\nDhbmmbDWdHbs2NHoIgw7DhZmZlZVnx5RbvsnP2fHmkFHR0e5r+LSSy8tp8+YMcPXbx04WJhZUygG\nhc7OTtra2hpanuHGzVDmuzJrOqUF0Kx+HCzMrOn4Bqf+qgYLSUdKuk3SjyU9KOninD5G0mpJj0ha\nJWl04ZiFkjZIeljSGYX0aZLWS3pU0pJC+khJy/Mxd0k6aqC/qFW2ZMmS6pnMhhAPna2/WmoWu4AP\nR8TrgTcCF0k6FlgA3BoRxwC3AQsB8rKqs4DjgLOAKyWVpo5fBcyNiKnAVEln5vS5pCVapwBLgMsG\n5NtZTVauXFk9k9kQ4mu2/qoGi4jYFhHr8vYvgYeBI4FzgPacrR2YmbfPBpZHxK6I6AQ2AKdIGg8c\nHhFrc75rC8cUz3UDcPq+fCnrm9/+9reNLoJZn/iarb8+jYaSNAloBdYA4yKiC1JAkXREzjYBuKtw\n2JactgvYXEjfnNNLx2zK59otaYeksRGxvU/fxmpWHIZ4xx13lEeWeBiiDVW+Zhur5g5uSYeR7vrn\n5xpG98e/DuTjYPd64qGZmTVOTTULSQeSAsWyiPiPnNwlaVxEdOUmpidz+hZgYuHwI3NapfTiMVsl\njQBGVapVzJkzpzxsrqWlhdbW1vJdRemuw/vV94t3Yl/60pdoa2vb6+FsQ6m83vd+aXvGjBl86Utf\nKqcPlfI1635pu7Ozk97UtJ6FpGuBpyLiw4W0xaRO6cWSPg6MiYgFuYP7OuBUUvPSLcCUiAhJa4CL\ngbXATcDlEbFS0jzg+IiYJ2k2MDMiZvdQDq9nMUA6us2GXbRoEeAqvQ1dvmbro9J6FlWDhaQ3AXcC\nD5KamgL4BHAPsIJUI9gIzIqIHfmYhaQRTjtJzVarc/rJwFLgYODmiJif0w8ClgEnAU8Ds3PnePey\nOFgMgtbBTDGvAAAJnElEQVTWVg9FtKbia3bw9DtYDCUOFoPD//Cs2UyYMIEtW7ZUz2h9VilY+NlQ\nw1SxSv/AAw94ZIkNecVrduvWrb5m68w1C6Otrc0PZbOmcsghh/Cb3/ym0cXYL7lmYRVVGwVhNhQs\nWbKEG2+8EUiT8kq1iZkzZ3LJJZc0sGTDg2sWxlve8hY/PsGaSr77bXQx9kuVahZ+6qwxfvz4RhfB\nrKpzzz2XlpYWWlpaAMrb5557boNLNjy4GWqYKnYWtre3lyc6urPQhqr58+dz4oknAmmeRanpyddr\nfThYDFPFoHDjjTe6g9vMeuU+C2P8+PFs27at0cUwq5n7LAaPR0PZHorNUF1dXR6zbkPeBz/4Qb7z\nne+U90tNp29729u44oorGlSq4cPBwsyawuTJk8sBYuPGjeXtyZMnN65Qw4ibocyP+7Cm42aoweNm\nKKvo5z//eaOLYFZVsekUcNNpnXmehZmZVeWaxTDlh7KZWV84WJhZUyjeyHz605/23KA6qxosJF0N\nvA3oiogTctoY4OvA0UAnaeGjZ/N7C4ELgV3sufDRNPZc+OiSnD4SuBY4GXgKeHdEPDFwX9HM9gfF\n2vDu3btdG66zWvosrgHO7Ja2ALg1Io4BbgMWAuQlVWcBxwFnAVdKKvWqXwXMjYipwFRJpXPOJS3P\nOgVYAly2D9/HzPZT69at2yNglLY9kq8+qtYsIuL7ko7ulnwOMD1vtwMdpAByNrA8InYBnZI2AKdI\n2ggcHhFr8zHXAjOBVflci3L6DYBn19RB6R9eSWm7paXFd2k2JLW2trJjxw4A7rjjjvJ12tra2sBS\nDR/97bM4IiK6ACJim6QjcvoE4K5Cvi05bRewuZC+OaeXjtmUz7Vb0g5JYyNiez/LZjW45JJLyg9i\nGzly5B6Bw2wo8g1OYw1UB/dAzo7ZazKIDbziQjI7d+70QjI25Llm0Vg1zeDOzVD/WejgfhiYERFd\nksYDt0fEcZIWABERi3O+laQmpo2lPDl9NjA9Ij5QyhMRd0saAfwsIo7YuxRpBvcFF1xQnubf0tJC\na2tr+aIp3Wl4v+f9F7uP+qZ0jTS6/N4ffvtvfvOb6Y+IGBLlb4b90nZpxcz29vYeZ3DXGiwmkYLF\nG/L+YlKn9GJJHwfGRMSC3MF9HXAqqXnpFmBKRISkNcDFwFrgJuDyiFgpaR5wfETMy0FkZkTMrlAO\nP+5jEMyYMWOPC8dsqJs8eTKPPfZYo4uxX+r34z4kXQ/MAF4m6QlSTeFzwDckXUiqNcwCiIiHJK0A\nHgJ2AvMKv+4XsefQ2dI6nlcDy3Jn+NNAj4HCBs/MmTMbXQSzPjnttNMaXYRhp5bRUO+p8NafVsj/\nWeCzPaTfC7yhh/TnycHGGsNtvtZsfM3Wn58NZSxd2ugSmPXNunUOFvXmR5QbEvjPas3E1+zgqdRn\n4ZqFmZlV5WBhpAn4Zs2ko9EFGHYcLMzMrCr3WZjbf63p+JodPO6zGEbGjk3/mGp9Qd/yS+kzzAZC\nX69XX7ON4WCxH3rmmXTXVevr9ts7+pQ/In2G2UDo6/Xqa7YxHCzMzKwq91nsh+rRnus2Yxso9bqW\nfM3Wxn0WZmbWbw4W+6Ggbz1/HX3tKZTSZ5gNgL5er75mG8PBYj8k+txb2OceRg3oelc2nPX5evU1\n2xDus9gP9XONoz4ZMwa2e+FbGwD1uF7B12yt+r2ehTWfvsZTd/xZI/Xn2vM1W39DphlK0lsk/UTS\no3n1PaubjkYXwKyPOhpdgGFnSAQLSQcAVwBnAq8HzpN0bGNLNZysa3QBzPrI12y9DYlgAZwCbIiI\njRGxE1gOnNPgMg0jOxpdALM+8jVbb0MlWEwANhX2N+c0MzMbAtzBbZx4Ymeji2DWI/UyVEq6tOJ7\nHjU58IZKsNgCHFXYPzKn7aW3i8f6T2pvdBHMBox/JwbekJhnIWkE8AhwOvAz4B7gvIh4uKEFMzMz\nYIjULCJit6QPAqtJ/ShXO1CYmQ0dQ6JmYWZmQ9tQGQ1lDSDpBUn/VNj/iKS/b2SZzHoj6b8kvaWw\n/y5JNzeyTMOFg8Xw9jzwdklecNKaxd8CX5A0UtJhwGeAeQ0u07DgYDG87QK+DHy40QUxq0VE/Bj4\nNrAA+BTQHhGdDS3UMOE+i2FM0nPAq4AHgROAvwEOjYh/aGjBzHoh6aXAfaSa8e/lpz7YIBsSo6Gs\ncSLil0qTLOYDv2l0ecyqiYhfS/o68AsHivpxM5QBfBGYC7y00QUxq9EL+WV14mAxvAkgIp4BVgDv\nb2xxzGyocrAY3oodVp8HXtYtzcwMcAe3mZnVwDULMzOrysHCzMyqcrAwM7OqHCzMzKwqBwszM6vK\nwcLMzKpysDAzs6ocLGxYkPSLBnzmJyX9SNIDku6T9Ps5fb6kg2s4vqZ8ZvXgSXk2LEh6LiJG1fHz\n/oA0K356ROzKa4aMjIhtkn4KnBwR26uco6Z8ZvXgmoUNW5KOlvQ9Sesk3SLpyJz+NklrJN0rabWk\nV+T0RZKulnS7pMckfaiX078SeCoidgFExPYcKD5Eeiz87ZK+l897paR7JD0oaVFO6ylfuXYk6R2S\nrsnb78rH3i+pY4D/TGaAaxY2TPRUs5D0bWBFRHxV0vuAsyPiXEmjI+LZnGcucGxEfCz/kP8ZMAMY\nDTwCjIuI3T183qHA94FDgO8BX4+IO/N7j5NqDM/k/ZaI2CHpgJz3QxHxox7ylb+DpHcAfx4RF0pa\nD5wZET+TNCoinhvQP54ZrlnY8PZG4Gt5exlwWt6eKGlV/hH+KPD6wjE3RcSuiHga6ALG9XTiiPgV\nMI20oNTPgeWSzs9vK79KZku6F7gfeF1+9ZSvku8D7ZLej9eosUHiYGHDWfdqdWn/X4DLI+IE0prP\nxU7m5wvbL9DLj3Mkd0ZEG/Ah4B3d80iaBHwEeHNEnAjc3O3zKinniYh5wCeBicC9ksbUcLxZnzhY\n2HDR0x36D4Dz8vZfAv+Vt0cBW/P2Bf36MGmqpMmFpFZgY95+Ln9G6bN+CfxC0jjgrMIxxXwA2yQd\nk5urzi181msiYm1ELAKeJAUNswHlKqsNF4dIeoIUNAL4Auluf6mkj5Kait6X814K3CBpO3AbMKnC\nOXvr8DsM+BdJo4FdwGOkJimArwArJW2JiNMlrQMeBjaRmpToKR+wELiJFBB+mD8D4J8kTcnbt0bE\n+t7/FGZ95w5uMzOrys1QZmZWlZuhzPZBnmz3PV5skio1c51eGvJqtj9wM5SZmVXlZigzM6vKwcLM\nzKpysDAzs6ocLMzMrCoHCzMzq+r/A8sSuJNbeMdYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a4d1310>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "data.boxplot(column=\"ApplicantIncome\",by=\"Loan_Status\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x10d71ced0>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x10d770b90>], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEZCAYAAACXRVJOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGzlJREFUeJzt3X+QHPV55/H3B8koCIyQFdiNESAc/wJXEo4LIhfs8hJ8\ngHEKEf+BsV13+AdXqQIiXZzyIfmuSkvqLkY+OzaVFKniwJTMQRTB2QGnHJApMkmRsy0wyAgkZMWO\n+CGzC0IEHxiIBM/90V9Js7sz2p7p7pme2c+rampnvzP99NO9/ewz3T3To4jAzMzsiH4nYGZm9eCG\nYGZmgBuCmZklbghmZga4IZiZWeKGYGZmgBuCmZklbgg1JWmXpElJRzWNfVbS3/UzL7Nek3SrpK9P\nG/ugpD2SRvqV1zByQ6ivIPv7/OcW42ZzySrgQknnAUhaANwI/GFETPY1syHjhlBv/xP4I0nH9jsR\ns36JiL3ASuBGSQuBceCfIuLWviY2hNwQ6u0hoAF8vs95mPVVRNwJPAz8JXAF8J/6m9FwckOov7XA\n1ZKW9DsRsz67Cvgd4NqI+Fm/kxlGbgg1FxGPA38DrOl3Lmb9FBHPAXuAbf3OZVi5IQyGcbJd5BP7\nnIeZDTE3hAEQET8B/orsxJqZWSXcEOpr+ttL/xhY2GLcbC7x9l+hWRuCpJvTB6QebRpbLGmTpB2S\n7pW0qOmxNZJ2Stou6fyqEh92EfGOiLi/6fdnImJhRJzXz7zmMklLJd0v6XFJWyX9QRpfK+kZSQ+n\n24VN07geSjS9Lqxcmu0b0yS9H3gZ+EZE/HoaWwe8EBFfknQNsDgiVks6HbgNOAtYCtwHvCv8tWw2\nBCSNAqMRsUXSMcAPgRXAx4D/FxF/Ou35pwG343qwATHrHkJEPAC8OG14BbA+3V8PXJLuXwxsiIj9\nEbEL2AksLydVs/6KiImI2JLuvwxs59CJfrWYZAWuBxsg3Z5DOOHAR8YjYgI4IY2fCDzd9Lzd+J0x\nNoQkLQPOAH6Qhq6WtEXSTU2HUF0PNlDKOqnsXWCbM9LhojuBVWlP4QbgHRFxBjABfKWf+Zl1a36X\n001KGomIyXRc9bk0vhs4qel5S9PYDJLcRKwyEdHqEE5hkuaTNYNbI+KuNK/nm57yv4Bvp/uuB+u7\nTmoh7x6CmHqM9G7gU+n+5cBdTeOXSTpS0qnAO4HNh0m00G3t2rV9j1GHHLwcU28V+zqwLSKuPzCQ\nXhQd8FHgsXS/p/VQxbrsVdy5nmtVy9+pWfcQJN0OjAFLJD1Fdm2d64A7JH0GeBK4NG3Q2yRtJPto\n+T7gyugmK7MaknQO8Elgq6RHyA6VfgH4hKQzgDeBXcDvg+vBBs+sDSEiPtHmoQ+1ef4XgS8WScqs\njiLiH4F5LR665zDTuB5sYAz0J5XHxsb6HqMOOZQRow45lBXDMlWtyyrizvVc67Ldz/rBtMpmLHnv\n2SohiajopHJVXA9WhU5rYaD3EMzMrDxuCGZmBrghmJlZ4oZgZmaAG4KZmSVuCGZmBrghmJlZ4oZg\nZmaAG4KZmSW1agijo8uQNOU2Orqs32mZmc0Jtbp0hSRmfteOurqMq81dvnSFWcaXrjAzs664IZiZ\nGeCGYGZmiRuCmZkBbghmZpa4IZiZGeCGYGZmiRuCmZkBbghmZpa4IZiZGeCGYGZmiRuCmZkBbghm\nZpa4IZjV1PTLwftS8FY1X/7ahs6wXP56Zj24Fqwzvvy1mZl1xQ3BzMwANwQzM0vcEMzMDHBDMDOz\nxA3BzMyAgg1B0h9KekzSo5Juk3SkpMWSNknaIeleSYvKStasnyQtlXS/pMclbZW0Mo233eYlrZG0\nU9J2Sef3L3uz2XX9OQRJbwceAN4bEf8q6a+A7wCnAy9ExJckXQMsjojVLab35xCsElV9DkHSKDAa\nEVskHQP8EFgBfJoW27yk04HbgLOApcB9wLtmbPj4cwhWjV5/DmEecLSk+cBRwG6yAlmfHl8PXFJw\nHma1EBETEbEl3X8Z2E72j77dNn8xsCEi9kfELmAnsLynSZt1oOuGEBE/A74CPEXWCF6KiPuAkYiY\nTM+ZAE4oI1GzOpG0DDgD+D7tt/kTgaebJtudxsxqqeuGIOk4sldGpwBvJ9tT+CQzj/l4H9eGSjpc\ndCewKu0peJu3oTC/wLQfAn4aEXsBJH0L+G1gUtJIREymY67PtQswPj5+8P7Y2FiBVGwuazQaNBqN\nnswrHR69E7g1Iu5Kw+22+d3ASU2TL01jLbkerKiitVDkpPJy4GayE2avA7cADwInA3sjYp1PKls/\nVHlxO0nfAPZExOeaxtbRYptvOql8Ntmhou/ik8rWQ53WQqGrnUpaC1wG7AMeAa4A3gpsJHtl9CRw\naUT8S4tp3RCsEhW+y+gc4B+ArWQbagBfADbTZpuXtAb4LFmNrIqITW1iuyFY6XraEIpwQ7Cq+PLX\nZhlf/trMzLrihmBmZoAbgpmZJW4IZmYGuCGYmVnihmBmZoAbgpmZJW4IZmYGuCGYmVnihmBmZoAb\ngpmZJW4IZmYGuCGYmVnihmBmZoAbgpmZJW4IZmYGuCGYmVnihmBmZoAbgpmZJW4IZmYGuCGYmVni\nhmBmZoAbgpmZJW4IZmYGuCGYmVnihmBmZoAbgpmZJW4IZmYGuCGYmVnihmBmZoAbgpmZJW4IZmYG\nuCGYmVnihmBmZoAbgpmZJYUagqRFku6QtF3S45LOlrRY0iZJOyTdK2lRWcma9ZukmyVNSnq0aWyt\npGckPZxuFzY9tkbSzlQj5/cna7N8iu4hXA98JyJOA34DeAJYDdwXEe8B7gfWFJyHWZ3cAlzQYvxP\nI+LMdLsHQNJpwKXAacCHgRskqXepmnWm64Yg6VjgAxFxC0BE7I+Il4AVwPr0tPXAJYWzNKuJiHgA\neLHFQ63+0a8ANqTa2AXsBJZXmJ5ZIUX2EE4F9ki6Je0m3yhpITASEZMAETEBnFBGomY1d7WkLZJu\najpMeiLwdNNzdqcxs1qaX3DaM4GrIuIhSV8lO1wU0543/feDxsfHD94fGxsrkIrNZY1Gg0aj0c8U\nbgD+OCJC0n8HvgJc0WkQ14MVVbQWFNH2//XhJ5RGgO9FxDvS7+8nawi/CoxFxKSkUeDv0jmG6dPH\n9Hlnh1en5yO6zdHmJklERGXH6iWdAnw7In79cI9JWg1ERKxLj90DrI2IH7SYLkc9uBasM53WQteH\njNJhoaclvTsNnQc8DtwNfCqNXQ7c1e08zGpKNJ0zSC98Dvgo8Fi6fzdwmaQjJZ0KvBPY3LMszTpU\n5JARwErgNklvAX4KfBqYB2yU9BngSbJ3WZgNBUm3A2PAEklPAWuBcyWdAbwJ7AJ+HyAitknaCGwD\n9gFXztgNMKuRrg8ZFZ6xDxlZRao+ZFQFHzKyKvTskJGZmQ0XNwQzMwPcEMzMLHFDMDMzwA3BzMwS\nNwQzMwPcEMzMLHFDMDMzwA3BzMwSNwQzMwPcEMzMLHFDMDMzwA3BzMwSNwQzMwPcEMzMLHFDMDMz\nwA3BzMwSNwQzMwPcEMzMLHFDMDMzwA3BzMwSNwQzMwNgfj9nPjEx0c/Zm5lZE0VEf2YsxS/90vFI\n2U7K/v2vsG/fy8D0fES/crTBJImIUL/z6ISkmL6dS2JqPbgWrDOd1kJfDxm99tp2Xn11gldfnWDf\nvj/pZypmZnOezyGYmRnghmBmZokbgpmZAW4IZmaWuCGYmRnghmBmZokbgpmZAW4IZmaWuCGYmRlQ\nQkOQdISkhyXdnX5fLGmTpB2S7pW0qHiaZvUg6WZJk5IebRpru81LWiNpp6Ttks7vT9Zm+ZSxh7AK\n2Nb0+2rgvoh4D3A/sKaEeZjVxS3ABdPGWm7zkk4HLgVOAz4M3KDsAkVmtVSoIUhaClwE3NQ0vAJY\nn+6vBy4pMg+zOomIB4AXpw232+YvBjZExP6I2AXsBJb3Ik+zbhTdQ/gq8HmmXpJxJCImASJiAjih\n4DzM6u6ENtv8icDTTc/bncbMaqnrhiDpI8BkRGwBDrcb7Ov12lzjbd4GUpEvyDkHuFjSRcBRwFsl\n3QpMSBqJiElJo8Bz7UOsAxam+3sLpGJzWaPRoNFo9DOFyTbb/G7gpKbnLU1jLY2Pjx+8PzY2Vn6W\nNvSK1kIpX5Aj6YPAH0XExZK+BLwQEeskXQMsjojVLaYJ2AMsSSN/BqzEX5BjRVX9BTmSlgHfjohf\nS7+vA/ZO3+bTSeXbgLPJDhV9F3jXjG/CwV+QY9XotBaq+ArN64CNkj4DPEn2LguzoSDpdmAMWCLp\nKWAt2TZ/x/RtPiK2SdpI9i68fcCVrZqBWV309Ss0vYdgVfBXaJplBuorNM3MrD7cEMzMDHBDMDOz\nxA3BzMwANwQzM0vcEMzMDHBDMDOzxA3BzMwANwQzM0vcEMzMDHBDMDOzxA3BbGAsQNKU2+josn4n\nZUOkiqudmlklXmf6xR8nJwfqGn5Wc95DMDMzwA3BzMwSNwQzMwPcEMzMLHFDMDMzwA3BzMwSNwQz\nMwPcEMzMLHFDMDMzwA3BzMwSNwQzMwPcEMzMLHFDMDMzYCAawgJf7tfMrAcG4PLXUy/568v9mplV\nYwD2EMzMrBfcEMzMDHBDMDOzxA3BzMwANwQzM0vcEMzMDCjQECQtlXS/pMclbZW0Mo0vlrRJ0g5J\n90paVF66ZvUlaZekH0l6RNLmNOZ6sIFRZA9hP/C5iHgf8O+AqyS9F1gN3BcR7wHuB9YUT9NsILwJ\njEXEv4mI5WnM9WADo+uGEBETEbEl3X8Z2A4sBVYA69PT1gOXFE3SbECImTXlerCBUco5BEnLgDOA\n7wMjETEJWdMATihjHmYDIIDvSnpQ0hVpzPVgA6PwpSskHQPcCayKiJclxbSnTP/dbFidExHPSjoe\n2CRpBzO3f9eD1VahhiBpPlkzuDUi7krDk5JGImJS0ijwXPsI64CF6f7eIqnYHNZoNGg0Gv1Og4h4\nNv18XtJfA8vpoB7Gx8cP3h8bG6s2WRtKRWtBEd2/YJH0DWBPRHyuaWwdsDci1km6BlgcEatbTBuw\nB1iSRv4MWMnMF1CaNiaK5GzDTxIR0dOrIEpaCByR9pKPBjYB1wLnkbMepm/X0sxtv1V9uB6snU5r\noes9BEnnAJ8Etkp6hGxL/QLZy/6Nkj4DPAlc2u08zAbICPCtdMh0PnBbRGyS9BA56+Gb3/zmwfvH\nHXdcxemazVRoD6HQjL2HYBXpxx5CUZLi2GN/7+Dvr756L/v2/QLvIVgRPdtDMLNy/fznh/YQjjnm\nHezb9899zMbmIl+6wszMADcEMzNL3BDMzAxwQzAzs8QNwczMADcEMzNL3BDMzAxwQzAzs8QNwczM\nADcEMzNL3BDMzAxwQzAzs8QNwWygLUDSwdvo6LJ+J2QDzFc7NRtor9N8SezJyYG66rfVzADuIUx9\nReRXRWZm5RjAPYSpr4jAr4rMzMowgHsIZmZWBTcEMzMD3BDMzCxxQzAzM8ANwczMEjcEMzMD3BDM\nzCxxQzAbKv7gpnVvAD+YZmbt+YOb1r0h2UPwBb7MzIoakj0EX+DLzKyoIdlDMDOzotwQzMwMcEMw\nM7PEDcHMzIChbQizvxd7dHSZ369tZtZkSBvCgXcdHbpNTj455RnZ74d/jtkwavViaN68o/3iyKpr\nCJIulPSEpB9Luqaq+ZjVXf9rYeoec6sXQ2+++Qv84sgqaQiSjgD+HLgAeB/wcUnvLX9Ojb7HaDSK\n51CHGHXIoawYddK7WmilkX5O32MuGLWCv1FVf/dBybUu231VewjLgZ0R8WRE7AM2ACvKn02j7zHq\n8k/QDaG2elQLrTQKTDvzPNyBw0rnnntu6YeV3BDKj9mNqhrCicDTTb8/k8b6aOoGXqXpx2h9PPbw\nmtfXtddeO2zrrIa1kMfM83CHDiutpY6HlVqdG/nyl7/W77QGSl8vXbFgwTXAUQC88caP2L+/yrlN\nv+hXdU3h0DHaA7/7UhqHM3V9jQPjc3KdLVjwBwfvv/76nj5m0r3R0WVTGsXIyClMTOzqybwyUw+J\nvfJKddtRVcvaarmqXI/NFFH8mOKMoNJvAeMRcWH6fTUQEbGu6Tnlz9gsiYhadJQ8tZDGXQ9WiU5q\noaqGMA/YAZwHPAtsBj4eEdtLn5lZjbkWbJBUcsgoIt6QdDWwiew8xc0uAJuLXAs2SCrZQzAzs8Ez\npJ9UNjOzTrkhmJkZ0MO3naZPZ67g0HuwdwN39+N4qqS3AUTE3rkeow45lBGjjBx6pU61kFdV67eK\nuM61+7g92UNI12/ZQPbm/83pJuAv09vwOo33tgML3ME0J0vaIOl54AfAZknPpbFlcylGHXKoy3L0\nWtm10GYeHddHmziVrN8q4jrXkuJGROU34MfAW1qMH0n2sf48MU4mK6TngZ3APwHPpbFlOab/HvAx\nYF7T2DzgMuD7OXMYihh1yKEuy9HrWxm10CZuofro5fqtIq5zLakmu515h4k+AZzSYvwUYEcvFvZw\nxZa3EIclRh1yqMty9PpWRi20iVvFP65K1m8VcZ1rOXF78rZTSReSXfFxJ4eu63Iy8E7g6oi4J0eM\nnRHxrk4fa3rOBmAvsL4ph5OAy4FfjohLc+QwFDHqkENdlqPXyqiFNnEL1Ueb6SpZv1XEda4l1WQv\nGgIcvAzwcqaeSHswIt7IOX3Rfx5HAp+lxck8sg8LvZ4jh6GIUYcc6rIc/VC0FtrErOIfVyXrt4q4\nzrWkmuxVQyhqUIvfrBdcH1aGgWkIZZB0AXAJUwvmrk5204clRh1yKCNGGTlYe1Wt3yriOtcS6mmQ\nGkKRhZX0NeDdwDfIrkkPsBT4j2QnXFbNlRh1yKEuyzFMyv4nU9X6rSKucy0pbrdntHt9A74GfIfs\nXRPvT7fL0tj1Oab/cZtxkf9dMUMRow451GU5huVWtD56uX6riOtcy4k7SJeuuCgiLoqIDRHxQLpt\nAD4CXJRj+tckndVi/CzgtZw5DEuMOuRQRowychgWReujlarWbxVxnWsJcfv6jWkdek3SWRHx4LTx\nvAv7KeAvJL2VQ7tTJwEvpcfyGJYYdcihjBhl5DAsitZHK5+imvVbRdwqYlYVt4qYpcQdmHMIks4E\n/gJotbBXRcQPc8YZpekYa0RMdJHLUMSoQw5lxCgjh0FXVn20iV3J+q0irnMtFndg9hAi4mHg7CIL\nK0lknwg9MP18SZPRQVcclhh1yKGMGGXkMAzKqI9Wqlq/VcR1rsXjDkxDgGILK+l84AayT4juTsNL\ngXdKujIiNs2VGHXIoS7LMUzK/idT1fqtIq5zLSlut2e0e30Dzie7YNffAjel2z1p7Pwc02+nxUW+\ngFOB7TlzGIoYdcihLssxLLei9dHL9VtFXOdaTtxB2kO4HvhQROxqHpR0Ktlb606bZfr5HDq22mw3\n8JacOQxLjDrkUEaMMnIYFkXro5Wq1m8VcZ1rCXEHqSEUXdivAw8qu+ZL87VeLgNuzpnDsMSoQw5l\nxCgjh2FRxT+ZqtZvFXGdawlxB+ldRmuAS8mu7z59YTdGxBdzxDgduJiZ31S1rYM8hiJGHXIoI0YZ\nOQyDMuqjTdxK1m8VcZ1rCfU0KA0BXPxmh+P6sMK6PYExaDdgEXAd2ReU7AVeIDsJcx1w3FyKUYcc\n6rIcvvV+/VYR17mWE3dgLl0haZGk6yQ9IWmvpBckbU9jx+UIsRF4ERiLiLdFxBLg3DS2MWcawxKj\nDjnUZTmGQgn10UpV67eKuM61jLhVvsIo8wbcC1wDjDaNjaaxTTmmb/v1hId7bBhj1CGHuizHsNyK\n1kcv128VcZ1rOXEHZg+B7P2166Lpk5cRMRER68g+jDObJyX9F0kjBwYkjUi6hkMn4eZKjDrkUJfl\nGBZF66OVqtZvFXGdawlxB6khFF3YjwFLgL+X9KKkvUADeBvZuzPyqCLGiynGkh7mUdd10e1yNNJh\nkm5yGBZV/JMp42/cq7hl1FW/cq3Neh2YdxlJWgysJvuKwBPS8CTZVwReFxEv5ojxXrKPcn8/Il5u\nGr8w8n9D13IgIuJBSe8DLiT7FOB3OlqgqTFvjYj/UGD6D5B9R+/WyHfJiLOBJyLiJUkLydbrmcDj\nwJ9ExEs5YqwEvhURXb+iUfa1jx8nu+bOfZI+Cfw2sA24MSL25Yjxq8BHyd5i+QawA7g9In7ebV6D\nqIz6aBO3cM20iVt6HbWYR6G6ahOzo1prE6Nw/bWJW7wmB6UhHI6kT0fELbM8ZyVwFdlZ9zOAVRFx\nV3rs4Yg4M8d81gIfJvsQ0HfJNowG8O+BeyPif+SIcXeL4d8B7geIiItzxNgcEcvT/SvScv012eUL\nvh0R180y/ePAb0TEfkk3Aq8A/wc4L41/NEcOL6XpfgLcDtwREXtmm25ajNvI1uVRZFflPBr4VspD\nEXH5LNOvBH4X+Aeya/4/AvwL8HvAlRHR6CSfYZWnPtpMV7hm2sQtXEctYhauqzZxC9Vam5iF669N\n3MI12dXJi7rdgKdyPGcrcEy6vwx4iGwDB3gk53y2AvOAhcDPgWPT+FHAozljPAz8b2AM+GD6+Wy6\n/8GcMR5puv8gcHy6fzTZK5fZpt/enM+0x7bkzYHskOP5ZJ+CfJ7s2jmXA2/NGePR9HM+2avZeel3\n5VmfB/4e6f5CoJHun5z3bzoXbnnq4zDrt1DNHO7vVqSOWsQsXFdt4haqtTYxC9dfu1yL1uTAXLpC\n0qPtHgJG2jzW7IhIu7wRsUvSGHCnpFNSjDz2R8QbwC8k/STSYYmIeFXSmzlj/CawCvivwOcjYouk\nVyPi73NOD3BEOkRwBNk/xOdTHq9I2p9j+seaXjX+SNJvRsRDkt4NzHqYJomIeBPYBGyS9BayV30f\nB74MHJ9zOY4kK66FZO+j3gssoLPrIb2RpjkmJfZUymfOKKE+WimjZlopo46mK6OuWilaa62UUX+t\nFK7JgWkIZBv1BWTvqW0m4P/mmH5S0hkRsQUgIl6W9Ltk1//4tZw5/KukhRHxC+DfHkxAWgTk2pDT\nH+yrku5IPyfp/O+wCPgh2bKHpF+JiGclHUO+Qr0CuF7SfwP2AN+T9DTZyccrcuYwZT6RHe+/G7g7\nHRfN42ayD9HMIyvkOyT9FPgtskswzOYmsmu3/AD4ALAOQNLxZI1lLilaH62UUTOtFK6j6Uqqq1aK\n1lorZdRfK4VrcmDOIUi6GbglIh5o8djtEfGJWaZfSvbKZMYXhkg6JyL+MUcOCyLi9Rbjvwz8SkRs\nnS1Gi2k/ApwTEV/odNoWsRYCIxHxzzmffyzZpXHnA89ExGQH83p3RPy4u0ynxHk7QET8TNkHqD5E\ndohjc87p30d2Jc/HIuKJovkMqqL10SZm4ZppE7f0OmoRq7S6ahO/o1prE6Pr+msTr3BNDkxDMDOz\nag3S5xDMzKxCbghmZga4IZiZWeKGYGZmgBuCmZkl/x9zXNTjpBgZpwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a4d1410>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.hist(column=\"ApplicantIncome\",by=\"Loan_Status\",bins=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that income is not a big deciding factor on its own as there is no appreciable difference between the people who received and were denied the loan.\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html#pandas.DataFrame.hist | \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.boxplot.html#pandas.DataFrame.boxplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Cut function for binning\n",
    "\n",
    "Sometimes numerical values make more sense if clustered together. For example, if we're trying to model traffic (#cars on road) with time of the day (minutes). The exact minute of an hour might not be that relevant for predicting traffic as compared to actual period of the day like \"Morning\", \"Afternoon\", \"Evening\", \"Night\", \"Late Night\". Modeling traffic this way will be more intuitive and will avoid overfitting minute details which practically make little sense.\n",
    "\n",
    "Here I've defined a simple function which can be re-used for binning any variable fairly easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "low          104\n",
      "medium       273\n",
      "high         146\n",
      "very high     91\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Binning:\n",
    "def binning(col, cut_points, labels=None):\n",
    "    #Define min and max values:\n",
    "    minval = col.min()\n",
    "    maxval = col.max()\n",
    "    \n",
    "    #create list by adding min and max to cut_points\n",
    "    break_points = [minval] + cut_points + [maxval]\n",
    "    \n",
    "    #if no labels provided, use default labels 0 ... (n-1)\n",
    "    if not labels:\n",
    "        labels = range(len(cut_points)+1)\n",
    "    \n",
    "    #Binning using cut function of pandas\n",
    "    colBin = pd.cut(col,bins=break_points,labels=labels,include_lowest=True)\n",
    "    return colBin\n",
    "\n",
    "#Binning age:\n",
    "cut_points = [90,140,190]\n",
    "labels = [\"low\",\"medium\",\"high\",\"very high\"]\n",
    "data[\"LoanAmount_Bin\"] = binning(data[\"LoanAmount\"], cut_points, labels)\n",
    "print pd.value_counts(data[\"LoanAmount_Bin\"], sort=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More: http://pandas.pydata.org/pandas-docs/version/0.17.1/generated/pandas.cut.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 12. Coding nominal data\n",
    "\n",
    "Very often we come accross a case where we've to modify the categories of a nominal variable. This can be due to various reasons:\n",
    "    1. Some algorithms (like Logistic Regression) require all inputs to be numeric. So nominal varialbes are mostly coded as 0---(n-1)\n",
    "    2. Sometimes the same category might be represented in 2 ways. For eg temperature might be recorded as \"High\", \"Medium\", \"Low\", \"H\", \"low\". Here, both \"High\" and \"H\" refer to same category. Similarly, in \"Low\" and \"low\" there is only a difference of case.\n",
    "    3. Some categories might have very low frequencies and its generally a good idea to combine them\n",
    "\n",
    "Here I've defined a generic function which takes in input as a dictionary and codes the values using 'replace' function in Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before Coding:\n",
      "Y    422\n",
      "N    192\n",
      "Name: Loan_Status, dtype: int64\n",
      "\n",
      "After Coding:\n",
      "1    422\n",
      "0    192\n",
      "Name: Loan_Status_Coded, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Define a generic function using Pandas replace function\n",
    "def coding(col, codeDict):\n",
    "    colCoded = pd.Series(col, copy=True)\n",
    "    for key, value in codeDict.items():\n",
    "        colCoded.replace(key, value, inplace=True)\n",
    "    return colCoded\n",
    "\n",
    "#Coding LoanStatus as Y=1, N=0:\n",
    "print 'Before Coding:'\n",
    "print pd.value_counts(data[\"Loan_Status\"])\n",
    "data[\"Loan_Status_Coded\"] = coding(data[\"Loan_Status\"], {'N':0,'Y':1})\n",
    "print '\\nAfter Coding:'\n",
    "print pd.value_counts(data[\"Loan_Status_Coded\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar counts before and after proves the coding.\n",
    "\n",
    "More: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html#pandas.DataFrame.replace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End Notes:\n",
    "\n",
    "In this article, we coverd various functions of Pandas which can make our life easy while performing data exploration and feature engineering. Also, we defined some generic functions which can be reused for acheiving similar objective on different datasets.\n",
    "\n",
    "Also See: If you have any doubts pertaining to Pandas or Python in general, feel free to discuss with us.\n",
    "\n",
    "Did you find the article useful? Do you use some better (easier/faster) techniques for performing the tasks discussed above? Do you think there are better alternatives to Pandas in Python? We'll be glad if you share your thoughts as comments below."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
